Context Engineering for AI Agents

Lessons from building Manus - an AI agent framework that leverages in-context learning of frontier models

The Choice: End-to-End vs In-Context Learning

At the beginning of the Manus project, we faced a key decision:

• Train an end-to-end agentic model using open-source foundations
• Build an agent on top of the in-context learning abilities of frontier models

We call our manual process of architecture searching, prompt fiddling, and empirical guesswork "Stochastic Graduate Descent"

Design Around the KV-Cache

The single most important metric for a production-stage AI agent: KV-cache hit rate

Why it matters:

• Directly affects latency and cost
• Context grows with every step, output remains short
• Average input-to-output token ratio: ~100:1

Cached input tokens cost 0.30 USD/MTok vs 3 USD/MTok for uncached - a 10x difference!

Improving KV-Cache Hit Rate

Key Practices:

• Keep your prompt prefix stable - Even a single-token difference can invalidate the cache
• Make your context append-only - Avoid modifying previous actions or observations
• Mark cache breakpoints explicitly - When needed, account for potential cache expiration

If self-hosting models using frameworks like vLLM, ensure prefix/prompt caching is enabled

Mask, Don't Remove

As agents gain more capabilities, their action space grows more complex

Dynamic action spaces (like RAG-based tool loading) cause problems:

• Changes invalidate KV-cache for subsequent actions
• Model gets confused when previous actions refer to undefined tools

Function Calling Modes

We constrain action selection by masking token logits directly:

Consistent tool prefixes (browser_, shell_) allow easy enforcement of tool groups

Use the File System as Context

Modern LLMs offer 128K+ token contexts, but in agentic scenarios:

• Observations can be huge (web pages, PDFs)
• Model performance degrades beyond certain context lengths
• Long inputs are expensive, even with prefix caching

Compression strategies are always designed to be restorable

Manipulate Attention Through Recitation

Manus creates and updates todo.md files during complex tasks

Why this matters:

• Typical tasks require ~50 tool calls - a long loop
• Models can drift off-topic or forget earlier goals

This avoids "lost-in-the-middle" issues and reduces goal misalignment

Keep the Wrong Stuff In

Agents make mistakes - that's reality, not a bug

Common impulse: Hide errors, clean up traces, retry actions

Problem: Erasing failure removes evidence

Error recovery is one of the clearest indicators of true agentic behavior

Don't Get Few-Shotted

Few-shot prompting can backfire in agent systems

Language models are excellent mimics - they imitate patterns in context

Danger in tasks with repetitive decisions:

• Agent falls into rhythm, repeating similar actions
• Leads to drift, overgeneralization, or hallucination